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Abstract 

Occupation-specific assessment over the Internet was evaluated in a 
pilot project that addressed four objectives. One hundred nineteen 
students in seven classes/schools took an administrative office 
technology assessment given on demand over the Internet. Sixty-one 
students had taken the same assessment in paper form six to eight 
weeksearlier. Thescoresfor Internet administration of that subgroup 
were significantly higher (4.85) but exhibited similar variability and 
internal consistency reliability. Responsesto a background and reaction 
survey indicated few problemswith testing (accessing, downloading, 
taking, and submitting the test, plus understanding feedback). A 
majority of students responded positively to global evaluations (75% 
reported a positive experience; 62% preferred or greatly preferred 
Internet format over the traditional). Results suggest that the 
advantages of testing via the Internet outweigh the disadvantages. 
Strategi es an d tacti cs f or m ovi n g vocat i on al assessm en t to t h e I n tern et 
are discussed. 


It is increasingly mandated that educators demonstrate the results of instruction, given the 
proliferation of standards (content and performance) in education (Resnick & Wirt, 1996). The 
requirementto demonstrate results isespecially important in career and technical education given 
substantial federal, state, and local investments. TheOhio Vocational Competency Assessment 
(OVCA) is a competency-based, content valid system that helps to meet this requirement. 


Background 

The OVCA system measures specific occupational and general employability knowledge that 
result from career-technical instruction and related experiences. TheOVCA system consists of 
three components and is administered annually with results reported to students, teachers, and 
administrators Thefirst of three components is 39 occupation-specific assessments. The second 
and third OVCA components are both general. One is a cross-occupational assessment, 
Employability Skills, and theother isa battery consisting of three of four ACT (formerly American 
College Testing) WorkKeys assessments. The specific assessments vary with occupational 
taxonomies. For example, program completersin theAdministrativeOfficeT echnology curriculum, 
which is part of the Business cluster, were the focus of this project. I ndividual learners whose 
school districts elect to usetheOVCA system might take as many as five assessments during an 
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annual testing window. The particular instruments used would be the occupation-specific 
Administrative Office Technology and a cross-occupational Employability Skills (CETE) 
assessment, plusthreeWorkKeys (Applied M athematics, Reading for Information, and Locating 
Information). While this traditional assessment system exhibits good psychometric quality in 
termsof reliability, content validity, and reasonable benchmark scores, therearepotential advantages 
toasecurecomputersystem capableof operating at a distance. 0 ne way to realizethese advantages 
isthrough strategic use of the Internet. 

H ambleton (1989, 1996), D rasgow, 0 Ison, Keenan, M oberg, and M ead (1993) and Sands, Waters, 
and M cBride (1997) have advocated the use of computers in educational and organizational 
assessment. Sandsetal. (1997), for example, edited avolumethat explores the evolution of the 
Armed Services Vocational Aptitude Battery from a traditional, paper-pencil assessment to one 
that is largely administered by computer and uses adaptive programming to present itemsto test 
takers. D rasgow and Olson-Buchanan (1999) edited a book that examines innovations in 
computerized assessmentacrossmultipledomainsfe.g., personnel selection, scholastic assessment, 
and certification). Kingsbury and H ouser (1999) presented their development and implementation 
of computerized assessment for the Portland City Schools. Zakrzewski and Bull (1998) presented 
a description of their planning and implementation (1994-1997) of summative and formative 
assessments at a university in England. 


Internet Importance and Expansion 

M embersoftheeducational community areawareofthefundamental changesin communication 
and commerce that areoccurringasaresultoftheexplosiveexpansion of the Internet. Physically, 
the Internet and World Wide Web (WWW ) are a connected network of computers that enable 
communication and commerce at a d i stan ce ( H ahn & Stout, 1994; M aran, 1995;Thomas, 1995). 
It is conceptually, however, that the influence of the Internet is vast (Feldman & Krumenaker, 
1995; Institute for Information Studies, 1997). The Internet creates a fundamental shift in how 
the enterprise of education is conducted. Curriculum, administration, and assessment are all 
being influenced and that influenceisgrowing. 


T esting Over the Internet 

In 1997, the Pennsylvania Department of Education sponsored a pilot study to evaluate use of 
the Internet for test delivery. The Pennsylvania project is reported in detail in a technical report 
(Slivinski, Hardwicke, Kapes, Boyer, Ip, & Martinez, 1997) and in two articles (Bicanich, 
H ardwicke, Slivinski, & Kapes, 1997; Kapes, M artinez, Ip, Slivinski, & H ardwicke, 1998). The 
articles presented evaluation of several research questions. In brief, the Pennsylvania project 
involved two research phases and included 14 schools and over 370 students. The Vocational 
Technical Educational Consortium of States (VTECS), a consortium that provides members 
with competency-based, career-technical outcomestandards, curriculum resources, and assessment 
vehicles, was the source of the test items. The computer served asa"page-turner," which means 
that the computer presents and scores the test, but does not adapt the itemsto examinees based 
on their responses as in computer adaptive testing (CAT). The first phase consisted of students 
from Computer Repair and Computer Specialist curricula (N =160), who completed randomly 
generated sets of VTECS test items for self-assessment and instructional purposes. The second 
phase involved testing studentsin Child Care and Auto Body Repair vocational programs (N =360) 
using both paper-pencil and Internet formatsfor comparison purposes. Thedesign was a test- retest 
equivalency with control groups. In addition to the test performance comparisonsforthesecond 
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design, short attitudesurveyswerecompleted by students, teachers, and test administrators. The 
results indicated, first, no statistically significant differences in test performance across groups 
defined by demographic and special needs status. Second, the Internet provided a cost-effective 
alternative to thetraditional assessment format. Third, students preferred Internet testing by a 3- 
1 margin. Fourth, a qualitative evaluation of technical feasibility indicated no insoluble 
implementation problems. Finally, test security wasnot adversely affected by the Internet format, 
although this conclusion must be qualified by the fact that the VT ECS item bank itself is not 
totally secure. 

Purpose and Objectives 

Two potential motivationsto develop and evaluate testing systemsar eresponavenessto stakeholders 
(speed of scoring, printing/mailing cost savings) and responsivenesstoadvandngtechnology(u3Sof 
computers, the Internet, and evolution toward computer adaptive testing). There are a number 
of advantagesto assessing students in thismanner. Because assessmentspecialistsareinterested in 
methods that provide additional value to stakeholders (eg., more rapid feedback to test takers, 
cost reduction, and security enhancement), the present project investigated thefeasibility of testing 
over the I nternet. T he primary research objectives were: 

1. To examine the descriptive and psychometric characteristics of the test delivered via the 
Internet (mid-M ay), 

2. Tocompareperformanceon the I nternet format test to a paper-pencil version of the test that 
the students had taken (in M arch), 

3. To assess student attitudes about I nternet testing, and 

4. To evaluate technical feasibility. 

Limitations 

Given our smaller sample and choice of program, we were unable to examine the effects of 
various demographics on test score differences. We were also unable to have sufficient 
experimental control to permit counterbalancing of administration. 

M ETHODS 

We collaborated with a consulting organization on a pilot evaluation of use of the Internet to 
deliver vocational testing. The project was funded by the state's Department of Education and 
was conducted during April-M ay 1998. The project involved testing students from multiple 
schools using the standard paper-pencil Admi nistrative 0 fficeT echnology assessment converted 
to Internet delivery and administering a survey measuring background and reaction variables. 

Project D esign 

TheOhio project differed from the Pennsylvania project in some ways, such as the use of different 
test items. Also, the Ohio project was smaller than the Pennsylvania project in terms of schools 
and participants. FI owever, the crucial similarity was evaluation of use of the I nternet to deliver 
testing. Theobjectives, as previously expressed, were similar. T he consulti ng firm involved in the 
project produces Internet computer programming for educational usage, including assessment 
and curriculum applications. The Pennsylvania testing project used software written in PERL 
whilethe present project, dueto security concerns, was written and implemented using aJAVA 
applet approach. JAVA is a programming language and applets are small programs that can be 
used as building blocks. 
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Internet Testing Procedures 

Thegeneral sequenceof Internet testing isasfollows. First, examineesaccesstheWebsiteoperated 
by the provider and choosewhetherto takeatutorial (which could have been taken anytimeprior 
to testing). W hen the student indicates her or his readiness, the complete test is downloaded 
from theserver. Theindividual movesthroughthetestrespondingtotheitemsand ispermitted 
to review responses. When ready, thetest issubmitted to the central server for scoring and return 
of feedback to the test-taker. For this study, the computer was considered a "page-turner" to 
administer and score the assessment. This usage corresponds to what FI ambleton (1996) termed 
'the earlier use of computersin assessment', but weassert that the benefits still outweigh thecosts. 

Security Issues and Solutions 

Security is a crucial issuefortheOVCA system, given the investment in its development and its 
high-stakes nature. We suggest that the best approach to test security is to use multiple methods 
of verifying identities and monitoring test-takers. Obviously, passwords for test takers are a 
fundamental attribute of computerized test security. We used a database program to create 
randomized passwordsthat conveyed information about schools, teachers, and students in 3 sets 
of 4-digits. 0 ne useful feature of the system isthatitcan be set up so that each password isonly 
valid onetime. An anecdote illustrates the utility of this feature. At one of the pilot schools, a 
teacher distributed the passwords in advanceofthetestingsession becausehebelieved (incorrectly) 
that a password was required to access the on-line tutorial. On the day of testing, three students 
reported that their password did not work, which suggests possible attempts to access the testing 
site. D uring larger-scale usage of thesystem, it will be possible to record the location from which 
the Website was accessed and evaluate such hypotheses. 

A second security concern for a distributed testing system pertains to the capability of test takers 
or others at a remote site to either print or save test materials from their computer stations. 
Proctors can help to eliminate some such problems, but there is a cost tradeoff associated with 
increased personnel. JAVA appletsenabled theavoidanceof theseproblemsby temporarily disabling 
the print and save features of the browsers (eg., Netscape) being used during testing sessions. 
Also, it ispossibleto store the location (Internet Portal address) from which an individual accesses 
the system in order to provide an "electronic trail." Finally, a third method of enhancing test 
security was the use of "windows" during which theserver would permit access from specified 
schools. T hus, schoolssigned up for primary and alternate windows T hus, security was approached 
in this project using multiple methods to tradeoff strengths and weaknesses. 

Administrative Office T echnology (AOT) Assessment 

The instrument used was identical to the paper-pencil version that had been administered as part 
of thestandard testing window during M arch-April 1998. Thesameform wasused to providea 
comparison across formats, although thedesign control to makethesamecomparisonsmadein 
the Pennsylvania project was absent. The 100-minute assessment contained 100 four-option 
multiple choice items, including seven items that involved graphics (e.g., tables or charts). 
Descriptive and psychometricfeaturesofthepaper-pencil assessment from 1995-1998 are presented 
in the results section. 

T raining for T eachers and T est Administrators 

Approximately one week before thescheduled testing window of M ay 7-15, 1998, weconducted 
atraining session for participating schoolsat a local Joint Vocational School. Theauthors, personnel 
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from the state Department of Education, and the consultant were present. Atthissession, held in 
a computer lab, we presented the rationale for Internet testing, a general overview of the system 
(including security issues), and details of the specific project. All attendees used a computer to 
access theWebsite and worked through an early version ofthetutorial and a practice test. Several 
suggestions were made at the session and implemented by the consultant. Following training, 
several i ndividuals stated that they believed the process could be extremely helpful in improving 
assessment by speeding up the feedback process and permitting assessment later in the school 
year. 

School and Student Sample 

Eleven schools (eg., Joint Vocational Schools, Career Centers) indicated willingnessto participate 
at thestart of the project. Duetovariousproblems, mostly technological in naturefe.g., software 
firewallsprohibiting access), thefinal set consisted of seven schools. The participating institutions 
represented a geographic cross-section of the state. Both urban and rural schools were present in 
the sample, although urban/suburban schools predominated. The student sample consisted of 
141 studentswho participated in thestudy and were assigned passwords. Onehundred nineteen 
studentstookthecomputerized assessment (performancedata). H owever, only about 100 students 
completed thebackgroundand reactionssurvey (attitudinal data), perhaps because of its positioning 
at the end of the assessment. 

Data Analysis 

Several analyses were conducted to address the objectives of thestudy. Various statistical aspects 
were examined, including (a) item/test statistics for both paper-pencil and Internet versions, (b) 
responses to the attitudinal surveys by test takers, and (c) relationships of these variables with 
demographics D escriptiveand psychometric statistics were calculated for usable responses to the 
100-item AOT assessment and thel6-item attitude survey. Correlational analyses were performed 
toexaminetheassociationsamongtestscores(paper-pencil and Internet), attitudinal responses, 
and demographic variables. The objective pertaining to feasibility, specifically the logistical and 
technical issues surrounding implementation, was addressed through telephone interviews with 
teachers. 

Results 

Table 1 contains results for the paper-pencil assessment for 1995-1998, based on samples of 
3,000 test-takers each year (3,500 take the test each year). The forms for 1997 and 1998 were 
kept identical to facilitate reporting and the establishment of benchmark scores. 

Tablel 


Descriptive and Psychometric Features of the Paper-Pencil Assessment, 1995-1998 


Year 

M ean 

Range 

Std Deviation 

Cron bach a 

1995 

48.60 

0-87 

13.63 

.886 

1996 

54.01 

1-91 

15.46 

.927 

1997 

55.85 

0-93 

17.22 

.924 

1998 

58.28 

1-93 

16.82 

.920 


Descriptive data related to the first research objective are provided in Table 2 and the subsequent 
paragraph. The demographic distribution of this sample indicatesa majority of females (92.4%) 
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and Caucasians (88.1%). All but two persons reported English as their first language. 
Approximately two-thirds of the students reported being in a two-year vocational curriculum 
(67.2%). Specific sample characteristics in Table 2 were comparable to the demographic profile 
of 3,436 individualstaking the 1998 paper-pencil assessment. Asistypical with field data, there 
were missing responses for test items and for reaction survey questions. Given the small sample 
size, all available data were used for the analyses. 


Table 2 


C haract eristics of StudentsT aking the I nternet Assessment 


Variable 

Levels 

Present Sample 

1998 Program Population 

Gender 


M ale 

9(07.6%) 

402 (11.3%) 


Female 

110 (92.4%) 

3137 (88.2%) 

Ethnicity 


African-American 

11(09.2%) 

440 (12.4%) 


Euro-American 

105 (88.1%) 

2497 (70.2%) 


Other 

2(01.7%) 

54(01.5%) 

M ulti racial 

Vocational 

1(00.8%) 

59(01.7%) 

Program 


1st of 2-year 

10(08.4%) 

304(08.5%) 


2nd of 2-year 

70 (58.8%) 

1780(50.0%) 


1st of 1-year 

36 (30.3%) 

1187(33.4%) 


Adult 

3(02.5%) 

184(02.5%) 


Due to space limitations, thefollowing data with respect to the assessment itself ispresented in 
narrativeform. A majority (91%) of the sample attempted 90 or moreitems(which reduced the 
variance of this variable). Thenumber of right scores on thecomputer-administered version fell 
between 15 and 91 (N =119). The average score was 65.42 with a standard deviation of 13.63. 
The cut points for dividing the score distribution into quartiles were 58.3, 67.5, and 75.0. The 
estimated Cronbach's a was .90. These values are comparable with those presented for 1995- 
1998 in Table 1. The one exception was the mean score on the assessment, which exhibited an 
increaseof 4.4 pointsfor the I nternet group compared to their previous paper-pencil scores. This 
finding was not unexpected given that the I nternet assessment wasa retest in a different format at 
an interval of two months. 

To address the second research objective, scoreson the paper-pencil and computer formats were 
evaluated for a subset of the students taking the I nternet assessment. Not all students take the 
paper-pencil assessment because the decision to test is made by school districts. W e identified 61 
studentsoutof 119 (50.4%) who had item scoresforboth paper-pencil and I nternet assessments. 
For this subset, we calculated means, standard deviations, and Cronbach a internal consistency 
estimates for both administration formats. Theresultsof this comparison are presented in T able 
3 and additional data related to thesecond research obj ecti ve i s p resen ted in narrativeform in the 
subsequent paragraph. 
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T able 3 


Comparison of the Internet and Paper-Pencil Formats(N =61) 


Paper-Pencil 

1 nternet 

59.33 

64.18 

14.81 

13.93 

2.3 

15 

82 

85 

.91 

.90 


Again dueto space limitations, thefollowing data with respect to comparing performanceon the 
Internet format test to a paper-pencil version of the test is presented in narrative form. Another 
way to view thelnternet version isasa retest with an interval of nearly two months, thecorrelation 
between the two scores for the subsample was .60. A paired t-test indicated that the difference in 
means was statistically significant (t =-2.94, df =60, p =.005). A second t-test was conducted to 
compare the I nternet number of right scores of two groups, those who had taken (N =61) with 
thosewhohad not taken (N =60) theearlier paper-pencil assessment. The results of that test were 
not significant (t=l. 20, df=119, p=.23). The mean differenceof 2. 95 favored those without prior 
experience but was not statistically significant. 


Online Survey Results 

An associated on-linesurvey (seeAppendix) addressed thethird research objective. Thesurvey, 
answered in part by 101 individuals taking the test (complete data were available for 65), was 
informative about the sample and their reactions to the Internet testing process. The survey 
questions used either Yes-No, 3-point, or 5-point anchored responses to investigate background 
and experience, the I nternet testing process, and overall reactions. Nearly a third of the sample 
indicated that they had received some assistance from a facilitator. Four questions pertained to 
computer access and experience (e.g., computer at home, daily access at school, self-evaluation of 
prior computer experience, and prior Internet experience). Approximately half of the sample 
reported that they accessed a computer at home, while95% reported having daily access at school. 
Again, most individuals (87 or 95%) reported at least moderate experience with the computer, 
but far fewer (16%) reported moderate or great experience with thelnternet. 

With respect to theexperiences reported by studentson theday of testing, survey questions asked 
about accessing thesite, using the mouse to answer itemsor navi gate through thetest, reading the 
itemson thescreen, submitting the answersfor scoring, receiving and printing the results, having 
keyboarding problems, understanding thecomputer-provided results, and experiencing computer 
anxiety during testing. O nly 17 of 92 students(16%) reported major or minor problems accessing 
thewebsiteor downloading thetest, whereas95% reported no problemswith usingthemouseto 
answer itemsor navigate through thescreens. Twenty-one of 91 students (23% ) reported minor 
or major problems reading the items on the computer screens. This pattern was repeated for the 
items pertaining to submitting answers for scoring, receiving answers, and understanding the 
results, with 90% or more of the sample endorsing the most positive scale value for these three 
items. With respect to computer anxiety, 9% reported great or major anxiety (8 of 93), while 
20% indicated some anxiety (19 of 93). 
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Two items served as overall indicators of student reactions. One asked for a comparison with the 
paper-pencil format and theotherforaglobal evaluation oftheday'sexperience. Concerningthe 
overall experience of the day's testing, the mean on a 5-point scale was 3.87 (SD =1.22). Specifically, 
69 of 92 (75%) students reported that I nternet testing was good or great, 16 (17%) reported a 
neutral evaluation, 7 (8%) reported that I nternet testing was satisfactory but needed work, and 
only 1 (1%) student indicated that Internet testing was a disaster. Concerning a comparison of 
paper-pencil and Internet formats, the mean score was 3.92 (SD = 89). Specifically, 58 of 93 
(62%) students indicated that Internet testing was better or much better than the traditional 
format, 22 (23%) reported a neutral opinion, 8 (9%) reported that I nternet testing was satisfactory 
but preferred paper-pencil, and 5 (6%) indicated that paper-pencil testsweremuch better. H owever, 
the reader should notethatonly44 of the61 individualswith scoreson both assessments responded 
to thisquestion. Finally, the relationship between test scores, on theonehand, and overall reaction 
and format comparison, on theother, were .42 and .34. These results, taken together, indicatea 
positive overall evaluation of the testing experience and a positive comparison of the Internet 
format relative to the traditional format. Attitudinal reactions were positive for the I nternet 
format (although wedid not obtain focusgroup or interview data with students to explore their 
specific concerns). 

Thefourth research objective, to evaluate technical feasibility, was addressed through telephone 
interviewswith teachers. T he logistical and technical issuessurrounding implementation wereof 
specific interest. Thus far, two occupational areas, Automotive Body Repair and Child Care/ 
Guidance, have been investigated in addition to AdministrativeO fficeT echnology. A comparison 
between formats revealed slight performance differences. The reactions of teachers examined by 
telephone interview were mostly positive. The major problems teachers reported during the 
interviews were related to the technical details of I nternet assessment and to some problems with 
presentation and scrolling of items on the screen (e.g., items with graphics). 


Conclusions and Future Directions 

Alternative assessment formats and techniques are assuming greater importance as educational 
institutions position themselves to take advantage of technology. As evidence of this, a special 
featureofthejournal Techniques(M arch, 1998), entitled "PuttingAssessmenttotheTest" reviewed 
fiveinnovationsin assessment. Theinnovationsweretitled "I stherea best way to test?. Engineering 
a grade, Windows on progress, Show & tell, and Online testing. The current project, which 
evaluated occupation-specific assessment over the I nternet was successful as measured by several 
quantitative and qualitative indicators. The direct comparison (delayed because of a time lag in 
receiving thescored test data from a contractor, which itself supportsthe I nternet testing concept) 
indicated that thel nternet test scores were higher than thepaper-pencil version, which was expected 
because the same form of the test was used. Scores on the I nternet assessments were comparable 
to previous years in terms of average, variability, and internal consistency. T he attitude survey 
responses indicated very positive evaluations of the process of I nternet testing and correspondingly 
positive reactionsto theoverall experienceand preferencesfor the I nternet format. Qualitatively, 
teachers/facilitators reported that they and thestudents were impressed with therapidity of reporting 
of results viaan overall score and scoreson six scoring clusters (ranging from office equipment/ 
procedures to professionalism). Their positive reaction is easy to understand, given a two-week 
delay in receiving score reports from the contractor. Although thedelay isexacerbated by school 
districts that are I ate in returning completed assessments, incorrect entries on answer documents, 
and the contractor's need to produce reports, it can be frustrating for teachers and students. 
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Based on the resultsof thissmall pilot study, it appearsthat there is potential for futuretesting via 
the Internet. The policy and planning steps required for implementation at the state level should 
be carefully considered, but would depend on the overall strategy determined by the state's 
Department of Education. Zakrzewski and Bull (1998) used their experiences over a four-year 
period to provide guidance for institutional strategies when considering implementation. Their 
framework involves a consideration of three areas of possible contribution from technology: 
formative assessment, summative assessment, and computer-aided learning. They advocateusing 
the progression from summative assessment to formative assessment to computer-aided learning. 

W hatever the framework, many of the decisions are the province of the State's D epartment of 
Education. 0 ne strategy would involve a gradual rollout across the state. One example would be 
to exten d th e I n tern et testi n g con cept to th e remai n i n g A d m i n i strati ve 0 ff i ceTech n ol ogy p rogram s. 
Oneadvantageofthistactic would bethattheAOT curriculum makes extensive use of computers. 
Another tactic would beto extend the Internet testing concept to an entire cluster, say Agricultural 
Education, and then to proceed cluster by cluster to roll out the program. Whatever the strategy, 
on e i m portan t p rel i m i n ary com pon en t sh ou I d be a tech n ol ogy assessm en t to en su re that targeted 
Vocational Education Planning Districts (VEPD) possess thehardware and software required for 
computerized systems. Specifically, the software used in the present project was developed using 
a philosophy of the lowest common denominator, thus Internet access using browsers (e.g., 
Netscape) isthe basic requirement. In considering thetask of moving the state's vocational education 
programs toward Internet testing, three interrelated dimensions seem of primary interest for 
implementation and marketing. In order of importance as they appear to us, thedimensionsare 
costs and benefits to stakeholders, details of the implementation, and evaluation of effectiveness 
of the computerized testing modality. 

Further, it should be cl ear that wearenot recommending that all assessment in vocational education 
move immediately to a computer adaptive testing (CAT) format, or even to a computerized 
format. While there are advantages to CAT (Wilson, Genco, & Yager, 1985), there are still 
significant advantages in using the computer as a page-turner that do not require advanced 
psychometric theory, computer programming, or huge empirically-validated item pools. 
Specifically, the speed of feedback benefits both learners and instructors, the capability to assess 
later in theschool year benefits those sametwo groups, and the reduction in printing and shipping 
costs benefits administrators and test developers. For example, software that offers most of the 
features of the software system that we used, specifically database capability for test items and 
Internet server interface with security features, is available at fairly modest prices. Low cost 
makes the technology more readily available to many vocational education systems. A second 
option for those schoolsthat lack Internet access isto providea self-contained assessment system 
on a compact disk (CD) that can betaken to the schools. Given the low cost of creating CDs 
(e.g., using CD-R and CD-RW peripherals), cost is not an issue. Also, it should be remembered 
that there are somedimensionsofthecareer-technical curriculum that computers cannot capture 
well at present, for example psychomotor skills. H owever, psychomotor dimensions are not 
impossible to assess but merely more difficult to program and might also require additional 
equipment for responding. For example, we maintain that the use of digital drawing tablets 
would permit theassessment of manual dexterity ( i . e. , in personnel selection) or artistic renderings 
(i.e., for graphic artists). 

Resultsof this pilot project, similar to those of the Pennsylvania project, indicate that Internet 
testing performscomparably and is preferred by studentsover thetraditional paper-pencil format. 
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Even though the present usage of the computer (page-turner) corresponds to what H ambleton 
(1996) termed 'the earlier use of computers in assessment,' it appears that this usage can be 
efficient and effective. Thus, we suggest that the advantages of Internet testing outweigh the 
disadvantages and that continued experimentation with this medium will create improvements 
that resolve those problems In sum, this medium can migrate from "vision to reality" (Bicanich 
et al., 1997). 
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Appendix 

Print-Based Copy of OnlineSurvey 


Ohio Pilot Project Evaluation: Thank you very much for taking this Internet-based test of your 
occupational specialty. In order to help us evaluate and improve our computer-based testing, 
please answer each of the questions below about your test experience today. After you have 
completed this evaluation, please push the SEN D EVALUATIO N button at the bottom of this 
form. 

1. If you were assisted in your test by a facilitator, check this bullet, o 

If no facilitator, check this bullet. o 

Prior Computer and Internet Experience 

2. Do you have a computer at home? 

Yes, I have a computer at home. 

N o, I do not have a computer at home. 

3. Do you have daily access to a computer at school? 

Yes, I have daily access to a computer at school. 

No, I do not have daily use of a computer at school. 

4. H ow would you rate your prior computer experience? 

No prior experience. 

Little prior experience. 

M oderateexperience. 

Lots of experience. 

A real computer pro. 

5. How would you rate your prior Internet experience? 

No prior experience. 

Little prior experience. 

M oderateexperience. 

Lots of experience. 

Real computer pro. 

Today's Internet Test Experience. 

6. D id you have any problems accessing our Website or getting your test downloaded 
M ajor problems getting thetest 

Someminor problems accessing or downloading thetest 
No problems 

7. Did you have any problems using the mouse to answer the questions? 

M ajor problems using the mouse 

Someminor problems using the mouse 
No problems 

8. Did you have any problems using the mouse to move from one question to the next? 

M ajor problems using the mouse 

Someminor problems using the mouse 
No problems 

9. Did you have any problems reading the questions on the computer screen? 

M ajor problems reading the questions 

Some minor problems reading the questions 
No problems 
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10. D id you have any problems submitti ng your answers and getti ng your test scored? 

M ajor problems with the scori ng 

Someminor problems with the scoring 
No problems 

11. Did you have any problems reading your test scores or printing them out? 

M ajor problems saving or printing test results 

Some minor problems saving or printing the results 
No problems 

12. To what extent did you have problems with keyboarding? 

M ajor problemswith keyboarding 

Great problemswith keyboarding 
Some problemswith keyboarding 
M inor problemswith keyboarding 
No problems 

13. Please look at your TEST RESULTS. Do you understand what each section means and 
how well you scored on the test? 

I do not understand the results 

I understand my basic percentage correct score, but do not understand the other results 
I understand my basic percentage correct score and overall results, but not my results by 
subject 

understand most of the results 
I understand all of the results 

14. To what extent were you anxious while working with computer technology? 

G reat anxieties 

M ajor anxieties 
Some anxiety 
Little anxiety 
No anxiety 

Compare today's Internet test with other paper-pencil tests 

15. H ow do you rate your overall experience today using the I nternet compared to the usual 
paper-pencil test? 

Paper-pencil tests are much better 

Although I like paper-pencil tests, the I nternet test was OK 

It makes no difference to me 

The I nternet test is better than the paper-pencil tests 

The I nternet test was much better than paper-pencil tests 

16. Overall, how do you rateyourexperiencetoday using thecomputer and I nternet for testing? 
It was a disaster 

It was OK, although I nternet testing needs a lot of work 
So-so 

T he I nternet testing was good 
T he I nternet testing was great 
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